Eye tracking method and system

ABSTRACT

A method for determining a series of gaze positions of at least one eye over time is provided. The method comprises capturing a video of a user&#39;s face simultaneously with displaying a stimulus video on a screen and extracting at least one color component for each one of a plurality of images obtained from the video of the user&#39;s face. Based on the at least one color component for each one of the plurality of images, the series of gaze positions of the user&#39;s face over the time of the video is determined. A system for determining a series of gaze positions of at least one eye over time is also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/894,476 filed Jun. 5, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/435,062 filed Jun. 7, 2019, which is acontinuation of U.S. patent application Ser. No. 16/283,360 filed Feb.22, 2019, which claims benefit from U.S. provisional patent application62/633,646 filed Feb. 22, 2018, the specification of which is herebyincorporated herein by reference in its entirety.

TECHNICAL FIELD

The present invention generally relates to eye tracking methods andsystems, and more particularly relates to gaze tracking methods andsystems using a camera and not requiring any other tracking device.

BACKGROUND

Eye movements are extremely fast and precise and remain largely intactin several neurological and medical conditions causing generalizedweakness and inability to speak and write. Such relatively commonconditions include strokes, amyotrophic lateral sclerosis (Lou Gehrig'sdisease), inflammatory polyneuropathies and intubation due to criticalillness, etc.

Communication with eye movements alone could also be valuable forvirtual reality, gaming, marketing and in various environments wherespeaking is not possible or desirable.

Several eye-tracking solutions have been proposed. Some of themtypically require the use of dedicated hardware such as infraredcameras, thereby reducing their availability and increasing the cost ofsuch technology. For example, eye tracking systems designed forparalyzed individuals are so expensive that they are unaffordable formost patients and clinical units.

A few experimental systems using ambient light only and ordinary webcamsexist but their performance is poor when lighting conditions aredegraded (dim/side lighting) or when the facial features of the user arenot strictly controlled (face not perpendicular to the camera, light eyecolor, dark skin color, etc.).

Capturing eye movements with an ordinary mobile device camera may bechallenging because of elements such as the many degrees of freedom ofsuch systems including parallel movements of the head and eyes, thechanging lighting conditions, the variability in eye and face shape andcolor, the sampling rate limitations of ordinary mobile device cameras,and/or the processor speed limitations of mobile devices.

Therefore, there is a need for an improved eye tracking method andsystem that may operate under ambient light conditions.

SUMMARY

According to an aspect of the invention, there is provided acomputer-implemented method for determining a gaze position of a user,comprising:

receiving an initial image of at least one eye of the user;

extracting at least one color component of the initial image to obtain acorresponding at least one component image;

applying a respective primary stream to each one of the at least onecomponent image to obtain a respective internal representation for eachone of the at least one component image;

determining an estimated gaze position for the initial image using therespective internal representation for each of the at least onecomponent image; and

outputting the estimated gaze position.

According to an embodiment, said applying the primary stream to obtainthe respective internal representation and said determining an estimatedgaze position are performed in combination and comprise:

processing the at least one component image using a neural network, atleast one portion of the neural network being the respective primarystream respective to each of the at least one component image, whereinthe neural network is implemented by one or more computers and comprisesone or more neural network layers, and wherein the neural network isconfigured to, at run time and after the neural network has beentrained, process the at least one component image using the one or moreneural network layers to generate the estimated gaze position.

According to an embodiment, said applying the respective primary streamto obtain the respective internal representation is performed separatelyfor each of the at least one component image, and comprises at least onefully-connected layer for each one of the at least one component image.

According to an embodiment, applying the respective primary stream toobtain the respective internal representation comprises at least oneconvolutional layer.

According to an embodiment, each respective primary stream comprises atleast one fully-connected layer downstream the at least oneconvolutional layer.

According to an embodiment, the neural network comprises anotherportion, namely an internal stream, downstream the respective primarystream, wherein said determining an estimated gaze position is performedusing the internal stream comprising at least one fusion layer, theinternal stream having at least one fully-connected layer.

According to an embodiment, the internal stream starts at a fusion layerreceiving at least the respective internal representation and furthercomprises an output layer downstream the fusion layer and comprising atleast one fully-connected layer.

According to an embodiment, said extracting at least one color componentcomprises extracting at least two distinct color components of theinitial image to obtain at least two corresponding component images, andfurther wherein said determining an estimated gaze position comprisescombining each of the respective internal representations together usingweight factors.

According to an embodiment, said extracting at least one color componentcomprises extracting each of three RGB components of the initial image.

According to an embodiment, the received initial image containsadditional features other than the at least one eye.

According to an embodiment, there is further provided:

identifying the at least one eye within the received initial image; and

extracting a portion of the initial image containing only the at leastone eye, thereby obtaining a cropped image,

wherein said extracting at least one color component is performed in thecropped image to obtain the corresponding at least one component image.

According to an embodiment, said identifying the at least one eye isperformed using a facial feature or landmark recognition method.

According to an embodiment, said identifying the at least one eyecomprises identifying at least one of: an outline of the at least oneeye, a position of at least one of a limbus, an iris and a pupil of theat least one eye.

According to an embodiment, said extracting at least one color componentcomprises extracting at least two distinct color components of theinitial image to obtain at least two corresponding component images, themethod further comprising:

for each of the at least two corresponding component images, determiningan illuminant value representative of the relative contribution of theeach of the at least two corresponding component images to the initialimage,

wherein said determining an estimated gaze position further comprisescombining the illuminant values with the respective internalrepresentation of the at least one component image.

According to an embodiment, the illuminant values are processed using anilluminant neural network comprising one or more fully-connected neuralnetwork layers.

According to an embodiment, the received initial image further containsat least one facial landmark, the method further comprising:

extracting the at least one facial landmark to obtain a corresponding atleast one landmark position;

wherein said determining an estimated gaze position further comprisescombining the at least one landmark position with the respectiveinternal representation of the at least one component image.

According to an embodiment, said combining is performed using a landmarkneural network comprising one or more fully-connected neural networklayers.

According to an embodiment, said determining an estimated gaze positioncomprises independently determining each of a first coordinate and asecond coordinate of the estimated gaze position.

According to an embodiment, there is further provided:

receiving at least one calibration image of the user associated with acalibration position; and

determining and outputting a calibrated estimated gaze position based onsaid at least one calibration image.

According to an embodiment, said determining and outputting a calibratedestimated gaze position comprises determining each of a first coordinateand a second coordinate independently.

According to an embodiment, said determining and outputting a calibratedestimated gaze position is performed using one of: a calibration neuralnetwork comprising one or more fully-connected neural network layers, aridge regression, decision trees, a support vector machine, and a linearregression.

According to an embodiment, there is further provided:

determining an orientation of the initial image relative to a reference;

wherein said determining an estimated gaze position are performed for apredetermined one orientation of the initial image.

According to an embodiment, there is further provided acquiring theinitial image of at least one eye of the user using a camera.

According to an embodiment, there is further provided determining theestimated gaze position relative to a screen of an electronic deviceusing a referential transformation, comprising querying screenproperties of the electronic device for performing the referentialtransformation, and performing user interactions with the electronicdevice based on the determining the estimated gaze position relative tothe screen of the electronic device.

According to another aspect of the invention, there is provided acomputer program product for determining a gaze position of a user, thecomputer program product comprising a computer readable memory storingcomputer executable instructions thereon that when executed by acomputer perform the method steps of any one of the methods above.

According to another aspect of the invention, there is provided a systemfor determining a gaze position of a user, the system comprising acommunication unit for at least one of receiving and transmitting data,a memory and at least one processing unit configured for executing themethod steps of any one of the methods above.

According to another aspect of the invention, there is provided a systemfor determining a gaze position of a user, comprising:

an extracting unit configured for receiving an initial image of at leastone eye of the user and extracting at least one color component of theinitial image to obtain a corresponding at least one component image;

an internal representation determining unit configured for applying arespective primary stream to each one of the at least one componentimage to obtain a respective internal representation for each one of theat least one component image; and

a gaze position estimating unit configured for determining an estimatedgaze position in the initial image according to the respective internalrepresentation of each of the at least one component image andoutputting the estimated gaze position.

According to an embodiment, the internal representation determining unitand the gaze position estimating unit are part of a neural networkimplemented by one or more computers and comprising one or more neuralnetwork layers, and wherein the neural network is configured to, at runtime and after the neural network has been trained, process the at leastone component image using the one or more neural network layers togenerate the estimated gaze position.

According to an embodiment, the neural network comprises:

at least one first neural network stream, each being configured togenerate the respective internal representation; and

a second neural network stream configured to generate the estimated gazeposition.

According to an embodiment, each of the at least one first neuralnetwork stream comprises at least one fully-connected layer.

According to an embodiment, each of the at least one first neuralnetwork stream comprises at least one convolutional layer.

According to an embodiment, each of the at least one first neuralnetwork stream further comprises at least one fully-connected layerdownstream the corresponding at least one convolutional layer.

According to an embodiment, the second neural network stream comprisesat least one fusion layer, each of the at least one fusion layer havingat least one fully-connected layer.

According to an embodiment, the second neural network stream furthercomprises an output layer downstream the at least one fusion layer andcomprising at least one fully-connected layer.

According to an embodiment, the extracting unit is configured forextracting at least two distinct color components of the initial imageto obtain at least two corresponding component images, and wherein saidgaze position estimating unit is configured for combining each of therespective gaze positions together using weight factors.

According to an embodiment, the extracting unit is configured forextracting each of three RGB components of the initial image.

According to an embodiment, the received initial image containsadditional features other than the at least one eye.

According to an embodiment, the extracting unit is further configuredfor:

identifying the at least one eye within the received initial image;

extracting a portion of the initial image containing only the at leastone eye to obtain a cropped image; and

extracting the at least one color component of the cropped image toobtain the corresponding at least one component image.

According to an embodiment, the gaze position determining unit isconfigured for identifying the at least one eye using a facial featureor landmark recognition method.

According to an embodiment, the gaze position determining unit isconfigured for identifying at least one of: an outline of the at leastone eye, a position of at least one of a limbus, an iris and a pupil forthe at least one eye.

According to an embodiment, the extracting unit is further configuredfor, for each of the component images, determining an illuminant valuerepresentative of the relative contribution of each one of the at leastone component image to the initial image, and wherein said gaze positionestimating unit is configured for combining each illuminant value withthe respective gaze positions.

According to an embodiment, there is further provided an illuminantneural network comprising one or more fully-connected neural networklayers and configured to process the illuminant values and generateilluminant data for transmission to the gaze position estimating unit.

According to an embodiment, the received initial image further containsat least one facial landmark, and wherein the extracting unit is furtherconfigured for extracting the at least one facial landmark to obtain acorresponding at least one landmark position, and further wherein thegaze position estimating unit is further configured for combining the atleast one landmark position with the respective gaze positions.

According to an embodiment, there is further provided a landmark neuralnetwork comprising one or more fully-connected neural network layers andconfigured to process the at least one landmark position and generatelandmark data for transmission to the gaze position estimating unit.

According to an embodiment, the first and second neural network streamsare configured to process each of a first coordinate and a secondcoordinate of the estimated gaze position independently.

According to an embodiment, there is further provided a calibrationmodel comprising one of: a calibration neural network, a ridgeregression, decision trees, a support vector machine, or a linearregression, configured to determine and output a calibrated estimatedgaze position.

According to an embodiment, the calibration model is configured toprocess each of a first coordinate and a second coordinate of thecalibrated estimated gaze position independently.

According to an embodiment, there is further provided an imageorientation determination module for determining an orientation of theinitial image relative to a reference, and wherein the gaze positionestimating unit each comprises four orientation modules, each beingconfigured for processing the initial image for a predetermined oneorientation of the initial image.

According to an embodiment, there is further provided a camera foracquiring the initial image of at least one eye of the user.

According to an embodiment, there is further provided an electronicdevice having a screen to which estimated gaze position relates.

According to an embodiment, the electronic device comprises the camera.

According to an embodiment, the electronic device comprises the computerprogram product, described above, installed thereon.

According to an embodiment, the computer program product is operated inthe electronic device to perform user interactions with the electronicdevice based on the gaze tracking.

According to another aspect of the invention, there is provided acomputer-implemented method for determining a gaze position of a user,comprising:

receiving an initial image of at least one eye of the user;

extracting at least one color component of the initial image to obtain acorresponding at least one component image;

for each one of the at least one component image, determining arespective gaze position;

determining an estimated gaze position in the initial image according tothe respective gaze position of each of the at least one componentimage; and

outputting the estimated gaze position.

According to an embodiment, said determining a respective gaze positioncomprises performing a regression method.

According to an embodiment, said determining an estimated gaze positioncomprises performing a regression method.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features and advantages of the present invention will becomeapparent from the following detailed description, taken in combinationwith the appended drawings, in which:

FIG. 1 is a flow of chart illustrating a method for determining a gazeposition of a user, according to one embodiment;

FIG. 2 shows the effects of head rotation on the projections of faciallandmarks, according to one embodiment;

FIG. 3 illustrates the decomposition of an image comprising 9 pixelsinto three component RGB images, according to one embodiment;

FIG. 4 shows an example of contrast between eye colors and sclera inindividual color channels of an RGB image, and between their grayscaleequivalents, according to one embodiment;

FIG. 5 is a schematic block diagram illustrating a regression algorithmused for implementing the method shown in FIG. 1, according to oneembodiment;

FIG. 6 illustrates the resizing, flattening and concatenation of twoimages, in accordance with an embodiment;

FIG. 7 illustrates the resizing, flattening and concatenation of twoimages, in accordance with another embodiment;

FIG. 8 is a schematic block diagram illustrating a system fordetermining a gaze position, in accordance with one embodiment;

FIG. 9 is a block diagram illustrating a processing module adapted toexecute at least some of the steps of the method of FIG. 1, inaccordance with one embodiment;

FIG. 10 illustrates the structure of an artificial neuron of a neuralnetwork;

FIG. 11 illustrates the structure of a fully-connected layer of a neuralnetwork, according to one embodiment;

FIG. 12 illustrates the structure of a convolutional layer of a neuralnetwork, according to one embodiment;

FIG. 13 illustrates the structure of a convolutional stream, accordingto one embodiment;

FIG. 14 illustrates the structure of a fully-connected stream, accordingto one embodiment;

FIG. 15 is a schematic block diagram illustrating an architecture usinga multi-layer perceptron for implementing the method of FIG. 1,according to one embodiment;

FIG. 16 is a schematic block diagram illustrating an architecture usinga convolutional neural network for implementing the method of FIG. 1,according to another embodiment;

FIG. 17 is a schematic block diagram illustrating the method of FIG. 1,wherein a calibration model is used, according to one embodiment;

FIG. 18 is a schematic block diagram illustrating the method of FIG. 1,wherein another calibration model is used, according to anotherembodiment;

FIG. 19 is a schematic block diagram illustrating the method of FIG. 1,wherein the calibration model has a vertical calibration model and ahorizontal calibration model, according to another embodiment; and

FIG. 20 is a detailed block diagram of an entire system for determininga gaze position of a user, according to one embodiment.

Further details of the invention and its advantages will be apparentfrom the detailed description included below.

DETAILED DESCRIPTION

In the following description of the embodiments, references to theaccompanying drawings are by way of illustration of examples by whichthe invention may be practiced. It will be understood that otherembodiments may be made without departing from the scope of theinvention disclosed.

FIG. 1 illustrates a method 10 for determining a gaze position of a userfrom an initial image, according to one embodiment. As it will bedetailed below, in one embodiment, the method 10 is executed by acomputer machine provided with at least a processing unit, a memory anda communication unit. The image of the user may be taken using a camerawhich may be integrated in a mobile and/or portable device such as asmartphone, a tablet, a phablet, a laptop, a computer machine providedwith a camera such as a webcam, or the like, or any dedicated deviceenabling to obtain images of the user. In one embodiment wherein acalibration procedure has to be performed, a display should be providedto the user, for example the display of the used mobile and/or portabledevice.

As it will become apparent below, in some embodiments, the method isimplemented in using neural networks. Neural networks are machinelearning models that employ one or more subsequent layers of non-linearunits to predict an output for a received input. Using neural networksconveniently trained enables to greatly improve the accuracy of thedetermination of the gaze position. The skilled addressee will howeverappreciate that simpler regression algorithms conveniently implementedmay be considered for specific applications, but accuracy of thedetermination of the position may not be sufficiently satisfactory, asdetailed below.

In the following description, the method and associated system fordetermining the gaze position of a user will first be described in abasic architecture using simple regression algorithms, according to someembodiments. More complex architectures using neural networks will bedescribed later with reference to FIGS. 15 to 20.

At step 12 of the method 10, an initial image of at least one eye of theuser is received. In one embodiment, the initial image comprises onlythe eyes of the user. In another embodiment, the received initial imagecomprises the two eyes of the user. In a further embodiment, thereceived initial image also comprises other facial features in additionto the eyes of the user, as detailed below. For example, the initialimage may comprise eyebrows, ears, a nose, a mouth, etc. In anotherembodiment, the initial image comprises the whole face of the user.

At step 14, at least one color component is extracted from the initialimage to obtain a corresponding at least one component image. In oneembodiment, two color components are extracted from the initial image toobtain two corresponding component images. In a further embodiment threecolor components are extracted from the initial image to obtain threecorresponding component images. Indeed, in one embodiment, the initialimage of the eye of the user is an RGB (Red-Green-Blue) image providedwith a red channel, a green channel and a blue channel. In thisexemplary RGB example, a single color channel is selected to build thecorresponding component image. More particularly, the decimal codeassociated with each pixel of the initial image received at step 12comprises a red value, a green value and a blue value. The red image isgenerated by taking into account only the red value of the pixels of theinitial image, i.e. the red image comprises the same array of pixels asthat of the initial image but the green and blue values of the pixelsare not taken into account so that only the red value of the decimalcode remains associated with each pixel. The red image represents thesame image as the initial image but only in red color. Similarly, thegreen image is generated by taking into account only the green value ofthe pixels of the initial image, i.e. the green image comprises the samearray of pixels as that of the initial image but the red and blue valuesof the pixels are not taken into account so that only the green valueremains associated with each pixel. The blue image is generated bytaking into account only the blue value of the pixels of the initialimage, i.e. the blue image comprises the same array of pixels as that ofthe initial image but the green and red values of the pixels are nottaken into account so that only the blue value remains associated witheach pixel.

As a result, in this example, the output of step 14 consists in thethree RBG component images, i.e. the red image of the eye of the user,the green image of the eye and the blue image of the eye.

It should be appreciated that the same extraction or decompositionprocess could also be applied to other color spaces, such as YCbCr, HSVor HSL for example. However, since the RGB color space is typically thecolor space in which colors are captured by digital cameras and storedin a computer, the RGB space may be preferred. The use of other colorspaces would indeed require an additional processing step to transformthe RGB value into the chosen color space. The method is applicable forimages collected using color components, such as RGB or othersubstantially equivalent color components, as described herein. However,the method could be applied under light conditions that would includelight components which are not visible, for example using infraredimages. Even though the method described herein does not requireinfrared projectors and cameras, the method can be applied to imagescomprising a component outside the visible spectrum. It should howeverbe noted that in infrared light conditions, the difference betweensclera and iris is very hard to identify as both appear grey in theimages, and using infrared is therefore not particularly advantageous.

At step 16, the respective gaze position for each of the at least onecomponent image is determined. It should be understood that any adequatemethod or algorithm for determining the gaze position may be used, asdetailed below. As a result, in the example using the three RGBcomponent images, a first gaze position is determined for the redcomponent image, a second gaze position is determined for the greencomponent image and a third gaze position is determined for the bluecomponent image. In the embodiment in which a single component image isused, a single gaze position will be determined at this step 16. Insteadof a respective gaze position, the component image may instead betreated individually by a respective primary stream (such as arespective portion of a larger neural network having convolutionallayers) which is used to obtain a respective internal representation. Aninternal representation is the output, within a neural network, of agiven layer of the neural network which is not the output layer.

At step 18, an estimated gaze position in the initial image isdetermined according to the respective gaze position of each of the atleast one component image. In the embodiment in which a single componentimage is used, the estimated gaze position corresponds to the singlerespective gaze position determined at step 16.

In the embodiment in which at least two color components are extractedfrom the initial image, the determined at least two respective gazepositions are combined together using weight factors to obtain theestimated gaze position, using any adequate combination method, asdescribed below. In the example using an RGB image, three respectivegaze positions are combined together using weight factors to obtain theestimated gaze position.

The thus-obtained estimated gaze position is then outputted at step 20.For example, the estimated gaze position may be stored in memory forfurther processing.

It should be understood that the initial image may comprise therepresentation of a single eye or both eyes. It should also beunderstood that the initial image may comprise two images, i.e. a firstimage comprising a representation of a first eye and a second imagecomprising a representation of a second eye.

In an embodiment in which the initial image comprises at least oneadditional facial feature in addition to the eyes, the method 10 furthercomprises a step of cropping the initial image to generate a croppedimage having a reduced size with respect to the size of the initialimage and comprising a representation of the one or two eyes only (forexample, two cropped eye areas, forming a composite image by beingjoined together, thus effectively removing the upper area of the nose).In order to crop the initial image, the eyes are previously identifiedwithin the initial image and extracted. It should be understood that anyadequate facial feature recognition method may be used for identifyingthe eyes within the initial image. For example, this may be done byidentifying the outline of the eyes, determining the position of thelimbus (i.e., the sclera-iris boundary), and/or the iris and pupil ofeach eye, within the initial image, as known in the art. It should beunderstood that any adequate method for identifying eyes within an imagemay be used.

Once the eyes have been identified within the initial image, the portionof the image that comprises only the eyes is extracted from the initialimage to create the cropped image. It should be understood that the sizeof the cropped image may vary so that the cropped image may comprisemore than the eyes for example, while still having a size that is lessthan that of the initial image.

In one embodiment, the Constrained Local Model (CLM) method is used foridentifying the eyes within the initial image. This method uses a numberof expert detectors each trained to recognize a specific facial featuresuch as the inside corner of the right eye or the bridge of the nose.Given the image of a face, each of these experts will produce anestimation of the location of the feature they were trained to detect.Appropriate locations are then connected to produce an outline of theanatomical features of the face. Commonly detected features include: theeyes, the eyebrows, the bridge of the nose, the lips and the jaw. Theears are also sometimes detected. By using the position of differentpoints relative to one another, a three-dimensional model of the facecan be constructed.

In one embodiment, the cropping of the initial image for isolating theregion of interest, i.e. the eyes, allows improving the signal-to-noiseratio of the data fed to the eye tracking algorithm (featureextraction), as well as decreasing the computational load(dimensionality reduction) and reducing the memory requirements forstoring data.

In one embodiment, the extraction of the eyes from the initial imageallows greatly reducing the input space to only contain relevant,non-redundant information.

As an example, assuming ideal western male facial proportions, and thatthe user's face is perfectly inscribed within the frame, the eyes willtogether represent about 40% of the horizontal space and about 7% of thevertical space of the initial image. This means that the images of botheyes together represent about 2.8% of the pixels of the initial image.The benefits are even greater if the user's face is smaller than theframe of the image. This allows reducing the demands for storage and thecomputational complexity of the below described regression problem, asfurther detailed below.

In a further embodiment, at least one additional facial landmark isextracted from the initial image in order to determine the head pose orattitude of the user in this image. In this embodiment, the at least oneadditional landmark is combined with the respective gaze positions todetermine the estimated gaze position. As it will become apparent below,such an embodiment enables to make the method more invariant to headpose.

Head pose is defined as the position of the head relative to the camera.This includes translation and rotation. As measured from an initialimage taken from a camera, translation would be measured of the distancebetween the center of the face and the center of the initial image.Rotation could be expressed in a number of ways, the most intuitive ofwhich, for a human, would be the Euler angles of the head, pitch (headnod), yaw (head shake) and roll (head tilt).

As previously mentioned, modern Infra-Red gaze tracking methods andsystems typically make use of a controlled source of light to estimatethe rotation of the eyeballs relative to the head, to then produce anestimate of gaze position. Such a system can thus be said to beintrinsically invariant to head pose.

On the contrary, the above described method of FIG. 1 does not make anydirect measurement of relative eye rotation, and so cannot be said to behead pose invariant. As previously mentioned, it is expected that themost relevant feature for estimating gaze position is the position ofthe limbus, or the boundary between the sclera and the iris, and theoutline of the eye. This changes when the head is fixed and the positionof the gaze changes, but also changes when the gaze is fixed and theposition of the head changes, either through translation or throughrotation.

Thus, in one embodiment, in order to produce more accurate gaze positionestimates, some information about head pose is added to the input dataof the method. As all features must be extracted from an image of theuser's face, the obvious candidate feature set for this is a set offacial landmarks whose positions relative to each other change as thehead moves and rotates. From these features, head translation can beeasily determined, for example by taking the distance between a fixedpoint on the image and a specific facial landmark, or between a fixedpoint on the image and the centroid of a set of facial landmarks.

The Euler angles of the head are much harder to estimate and require theprojections of the 2D coordinates of the facial landmarks onto a 3Dmodel of the user's face. Assuming that the model used is a perfectmodel of the user's face, the uncertainty on the angles would be thesame as the uncertainty on the positions of the facial landmarks. Giventhat the present method is meant to be deployed for use by the generalpublic, such an assumption cannot be made and a few models of the humanface need to be used instead, leading to an added uncertainty on theEuler angles.

In the context of training a machine learning algorithm, an idealfeature set should contain all the information necessary to solve theproblem, and only the information necessary to solve the problem. Bytransforming the coordinates of the facial landmarks into Euler angles,information about the topology of the face model is added to thefeature, which is relatively invariant through the dataset, whiledegrading the quality of the feature by increasing their uncertainty.For these reasons, the coordinates in image space of a set of faciallandmarks have been chosen to use as a feature to introduce head poseinvariance into our method.

It should be noted that such features already appear naturally in theeye images. Indeed, as the head moves and turns relative to the camera,the apparent height and width of the eyes also change. However, undernatural viewing conditions, the angle of the head relative to the camerawill hardly ever be greater than 30 degrees, at which point viewingbecomes uncomfortable. This means the apparent width and height of theeyes will nearly never vary by more than 15% of their maximum. Given theuncertainty in these measurements, this is unlikely to yield strong headpose invariance.

To better estimate head pose, in one embodiment, the XY coordinates ofcertain facial landmarks are used instead, provided that these landmarksdo not lie in the same plane in 3D space. This effect is illustrated inFIG. 2. Here, F₁, F₂ and F₃ could represent the positions of the lefteye, right eye and nasion, respectively, as seen from the top (thenasion being defined as the most anterior point of the frontonasalsuture that joins the nasal part of the frontal bone and the nasalbones, visible on the face as a depressed area directly between theeyes, just superior to the bridge of the nose). Two features could bechosen here: P₃, the length of the projection of the distance betweenthe eyes on the viewing surface, or P₁-P₂, the difference between thelengths of the projections of the distance between the left eye and thenasion, and the right eye and the nasion. The relationships between thevalues of those features and the angle of the head O is given byequations 1 and 2.

P ₃=2D ₁ cos(θ)  (1)

P ₁-P ₂=√{square root over ((H ₂ ∔D ₁ ²))}·(cos(θ−arctan(HD₁))−cos(θ+arctan(HD ₁)))  (2)

One immediate advantage of using P₁-P₂ over P₃ is that the formerpreserves information about the direction of rotation. Indeed, the valueof P₃ will always be positive for natural head angles, while P₁-P₂ willbe positive in one direction and negative in the other. Additionally, animportant aspect of a good feature is the difference in magnitudebetween extremes of the features. In other terms, a good feature shouldmaximize the difference between its minimum values and its maximumvalue. In this example, this will be the case if D₁<H, H being thedistance between the nasion and the eyes perpendicularly to the plane ofthe face and D₁ being the distance between the nasion and an eye in theplane of the face. In this example, the user's face is considered to besymmetrical, so D₂=2D₁. As it should now be apparent, a proper choice offacial landmarks can thus ensure these properties, making a choice offeatures that do not lie in a 2D plane much more interesting for headpose invariance.

Another advantage of using facial landmark coordinates over Euler anglesis that the facial landmark coordinates contain information about thedistance between the face and the camera, while the Euler angles do not.

Finally, it should be noted that depending on the chosen algorithm andarchitecture for performing the method, this information is not strictlyrequired for the model to perform well. However, if it is omitted,performance is expected to degrade quickly if the user moves his headaway from the typical position it was in during calibration, as it willbe detailed thereinafter.

FIG. 3 illustrates an exemplary decomposition of a color image 30 intoits RGB components. It should be understood that the image 30 may be theoriginal initial image or the cropped image as long as it contains theeyes.

The image 30 comprises nine pixels each having a different color. Eachpixel has a red value, a green value and a blue value associatedthereto, thereby forming the RGB components 32, 34 and 36 of the image30. The red component 32 comprises only the red value for the ninepixels of the image 30. The green component 34 comprises only the greenvalue for the nine pixels of the image 30. The blue component 36comprises only the blue value for the nine pixels of the image 30. TheRBG components are then isolated to create a red image 40 which includesthe nine pixels to which only the red value is associated thereto, agreen image 42 which includes the nine pixels to which only the greenvalue is associated thereto, and a blue image 44 which includes the ninepixels to which only the blue value is associated thereto.

It should be understood that each RGB component image corresponds to agreyscale image. Indeed, as the single-color image is a two-dimensionalmatrix such as a greyscale color image, the new single color image, i.e.the RGB component image, corresponds to a greyscale image, despiterepresenting a color channel. Thus, the greyscaling of the colorcomponents is simply a result of the decomposition.

It should be understood that in typical computer vision applications,images are normally fed as M×N×3 tridimensional matrices, comprising 3layers, each corresponding to one of the RGB components of the image.This matrix would typically be fed to the first layer of the network andtreated altogether in bulk (i.e., with the three layers, and using akernel or filter having the same depth), and the information related toeach of the RGB components will be “lost” in the following layers of thenetwork where all data are mixed into the subsequent layers. In such acase, it would not be possible to identify, at an internalrepresentation of the network, information specifically related to onecolor component only, as everything is already mixed starting at thefirst layer of the network being applied to the three-dimensionalmatrix.

Instead, in the present invention, the M×N×3 matrix is split in threedifferent two-dimensional matrices of M×N size (or M×N×1), and each oneis treated individually by its own portion of neural network (i.e.,their own distinct primary stream) before being fused after a few layersof their own distinct primary stream. For example, each of the threeM×N×1 matrices is fed to its own individual and distinct primary stream(portion of the neural network), which would comprise more than onelayer. For example, these individual and distinct primary streams foreach of the color component images could comprise 2 or 3 convolutionallayers and 2 or 3 fully-connected layers, before fusion. This ensuresthat information that can be found in a single color-component image iswell analyzed individually. The individual and distinct output of therespective primary stream for each color component image should not beconfused with the whole network's output (which can be trained), and itis rather called an internal representation of the network at that layer(to be fused in a step called feature fusion for further processingdownstream).

Making sure that the individual color component images are treatedaccording to their own, distinct primary stream has its advantages.Indeed, we have found empirically that depending on the circumstance,one of the color components (for example, in an RGB color space, one ofR, G or B) can be more appropriate or useful than the others. This canimprove accuracy, as described below. After applying in parallel thedistinct primary streams, all resulting internal representations fromthe three color component images (or more generally, from the at leastone color component image), are fused with the illumination informationand facial landmarks (or an internal representation thereof following anauxiliary stream). The conditions in which one of the color componentimages is more appropriate empirically depend on the illuminationinformation in the environment. There is no single color component whichis more adapted than another in every circumstance. Therefore, theneural networks adapt to the illumination context by performing a fusionbetween each color-component image (at the end of their own individualand distinct primary stream) and with the illumination information(which can also undergo an auxiliary stream). By doing this, the neuralnetwork automatically adapts to the real illumination context and usesthe most useful color component in this particular circumstance byperforming additional operations through subsequent layers of thenetwork, i.e., the internal stream, which is the portion of the neuralnetwork downstream of the fusion layer. In one embodiment, the mostrelevant feature for eye tracking in ambient light may be the positionof the sclera-iris boundary, or limbus, relative to the outline of theeye. Thus, a better contrast between the sclera and the iris would allowfor a better definition of this boundary and thus a more robust eyetracking method or algorithm. Different eye colors reflect differentamounts of red, green and blue light. For this reason, one can expectthat the identification of the limbus may depend on the user's eye colorand the ambient lighting conditions, and for the reasons describedabove, the neural network is trained to identify and use an internalrepresentation originating from a specific color component image (or aplurality thereof), for which the edges between sclera and iris, andbetween sclera and outline of the eye are more easily identified underspecific illuminant values, to be fed into the systems and combined withthe internal representation of the component images at the fusion layer.By decomposing the image into its RGB components, at least one of theresulting images may have a better contrast between the sclera and theiris. Thus, depending on the user's eye color and the temperature of theambient light, one of the three RGB component images should provide thebest contrast of the limbus. Moreover, we hypothesize that one of thecolor channels will always have higher contrast than in the equivalentgreyscale image. This is illustrated in FIG. 4, in which the contrastsbetween different eye colors under various lighting conditions, for eachof the RGB color channels and for the equivalent grayscale values, areillustrated. It is worth mentioning that, for each eye color andlighting combination, the greatest contrast between all the colorchannels is always greater than in the grayscale case.

The task of selecting which channel to prioritize is not a trivial one,as there exists infinite combinations of ambient lighting conditions andeye color. In one embodiment, a regression algorithm is used. While thecolor images could have been converted to grayscale, or the colorchannels concatenated to each other to be processed in the samepipeline, this would not have allowed the leveraging of thesedifferences between color channels. For this reason, the three colorchannels are processed separately, and then fused at the decision orfeature level, eventually using additional previously computed data suchas illuminant values, as described below.

While it is considered that having separate streams to process eachcolor channels separately is beneficial to the performance of themodel/algorithm, it is not necessary to include all three colorchannels. Indeed, considering that the fusion of the single-channelstreams is done through a weighted sum of each stream, which, whilebeing an oversimplification in the case of deep-learning models, is notinaccurate, the omission of one or more color channels would amount tosetting the weights applied to these channels in the weighted sum tozero. A model that only uses two channels or a single channel, or indeeda grayscale rendition of the color image, can be seen as a special casein which one or two processing streams are essentially ignored.

In one embodiment, as previously mentioned, the determination of therespective gaze position for the three component images is performedusing a regression algorithm/method. For example, linear regression,ordinary least squares, decision tree regression and/or artificialneural networks may be used.

In a further embodiment, the determination of the estimated gazeposition is also performed using a regression method or algorithm. Forexample, linear regression, ordinary least squares, decision treeregression and/or artificial neural networks may be used.

Regression algorithms usually follow a same training procedure. For thepurpose of the present description, the inputs are named X, theestimates are named Ŷ and the targets are named Y. In the present case,X would be the initial image of the user's eyes, Ŷ would be the estimateof the position of the user's gaze produced by the regression method,and Y would be the actual position of the user's gaze.

The training procedure creates a model F(X) that approximates amathematical relationship between X and Y, and that yields Ŷ from X. Inother words, Y≈Ŷ=F(X). The goal of the training procedure is to adjustthis mathematical relationship in a way to minimize the error between Yand Ŷ for any given X.

In the case of linear regression, F(X) may be expressed as:

F(X)=B+ΣW _(j) *X _(j)  (3)

where Xj is the jth feature of the input vector X, W_(j) is the weightassociated to that feature, and B is the Y-intercept, or bias, of thelinear regression model. In this case, the goal of the trainingprocedure would be to adjust the weights and the bias so as to minimizethe prediction error.

In one embodiment, regression algorithms also have hyperparameters,which affect the training procedure and therefore the final model, whichalso have to be optimized. In the present example of linear regression,the hyperparameter would tell whether or not to include a bias term inthe equation.

Hyperparameter optimization involves splitting the dataset into twoparts, the training set and the validation set. Prior to training, ahyperparameter search space is defined, which bounds the possible valuesof hyperparameters to be explored. For each set of values, the trainingprocedure described above is completed, and the performance of thetrained model is obtained from the validation set. The set ofhyperparameter values that yielded that best performance will finally beretained as the final model.

As described at step 18 of the method 10, the respective gaze positionsdetermined for the three RGB component images are combined together toprovide an estimated gaze position. It should be understood thatdifferent combination methods may be used.

In one embodiment, the estimated gaze position corresponds to a weightedaverage of the respective gaze positions determined for the three RGBcomponent images:

Ŷf=Σc*Ŷ  (4)

where Wc is the weight factor associated with each RBG component c.

In one embodiment, the weight factors are determined using a measure ofhow much each color channel contributes to the color image.

For example, the weight factors may be determined by calculating therelative contribution of each color channel by summing the values ofevery pixel of a color channel, and dividing the result by the sum ofall the pixels in the image. In one embodiment, such a method forcalculating the weight factors is simple, fast to compute and fairlyinvariant to light intensity. Indeed, lowering or increasing theintensity of ambient lighting would lower or increase the value of everypixel in every channel by a same factor, up to the point a pixel startssaturating. In one embodiment, the three values representing therelative contribution of each color channel correspond to the weightfactors Wc.

In another embodiment, a further regression algorithm may be used forcombining the three respective gaze positions obtained for the three RGBcomponent images. The inputs of the further regression algorithm couldbe the three values representing the relative contribution of each colorchannel and the three gaze positions obtained for the three RGBcomponent images, which would through training approximate therelationship between ambient light and color channel contribution.

As previously mentioned, in an improved gaze position estimation, thecombination of the three respective gaze positions obtained for thethree RGB component images could further been done as a function of theilluminant values representative of the relative contribution of eachcolor channel of the initial image.

In one embodiment, the illuminant values may be determining using themethod proposed in Yang, K. F., Gao, S. B., & Li, Y. J. (2015);Efficient illuminant estimation for color constancy using grey pixels;In Proceedings of the IEEE Conference on Computer Vision and PatternRecognition (pp. 2254-2263), but other methods may be used. For example,it may be considered to calculate the relative contribution of eachcolor channel by summing the values of every pixel of a color channel,and dividing the result by the sum of all the pixels in the image, aspreviously explained.

Other methods such as Gamut Constrained Illuminant Estimation and GreyPixel Illuminant-Invariant Measure may also be used, as it should beapparent to the skilled addressee.

Once the illuminant values have been determined, they are combined withthe respective gaze positions to determine an estimation of the gazeposition in the initial image.

FIG. 5 shows a regression algorithm used for implementing the method 10shown in FIG. 1, according to one embodiment. Three regressors aretrained as single channel regressors, each on a different color channelof the full color image of the user's eye. Their decisions are thencombined by a fourth regressor, also called prediction fusion, taking asan input the predictions from all three channels and the relativecontribution of each color channel to the image.

In this embodiment, four regression algorithms were tested assingle-channel regressors, that were deemed appropriate considering thefollowing parameters: small size of the initial dataset, low memoryrequirements and relatively low training time. These algorithms were:Ridge Regression, a Support Vector Machine (SVM), an ExtremelyRandomized Trees (ETR) and ElasticNet.

The image database used for training is collected from volunteers whowere asked to look at 13 predefined crosses on a computer screen. Eachcross appeared one after the other and stayed in view for three seconds.Subjects were given the first second to find the target. During the nexttwo seconds, ten images of the subject's face and surroundings werecaptured using a camera, to obtain images similar to those obtained froma mobile device's front facing camera. Then, the target disappeared andthe next target appeared. Ten images were captured for every cross toprovide usable data in the event of a blink.

To build the dataset used for training, the images containing thesubject's right and left eyes were cropped from the initial image usinga facial feature recognition algorithm to determine the location of theeyes and eyebrows in the initial image. This information was used todefine the bounding boxes for each eye, which were then used to crop theeyes. These two eye images were then associated with an (X, Y) set ofcoordinates representing the location of the center of the cross on thescreen at the time of image acquisition.

Referring now to FIG. 6, as the algorithms used in this embodiment onlyaccept one-dimensional matrices (i.e., vectors) of a fixed size asinputs, the eye images need to be resized and flattened before theycould be used. The resizing was necessary because there was no guaranteethat the cropped eye images would be the same size from frame to frame,or even as each other. Square crops were used to simplify the process,and both images were resized to be 25×25 pixels. This size was chosenempirically, as a compromise between an acceptable loss of resolutionand an increased size. The images are then flattened to make them onepixel high, while preserving the total number of pixels. Finally, theimages are concatenated to produce a single image with double the numberof pixels Finally, the images are concatenated to produce a single imagewith double the number of pixels. This image is the input to asingle-color regressor.

While the reshaped, concatenated and flattened eye images would besufficient to train an eye tracking system, the system would be verysensitive to head movements. To obviate this issue, a vector of (X, Y)facial landmark coordinates may also be concatenated to the eye vectorsto form the inputs to the algorithms, according to one embodiment and asillustrated in FIG. 7. In one embodiment, the XY coordinates of eightfacial landmarks are retrieved using a third-party facial landmarkdetection algorithm. These coordinates are flattened into a vector of 16values. After the processing steps described in FIG. 6, the eye vectorsare separated into individual color channels. Each of these vectors isthen concatenated with a copy of the facial landmark vector. Theresulting three vectors are finally used as the inputs to thesingle-channel regression algorithms.

Before training, a search space of possible hyperparameter values wasdefined for every algorithm under consideration. Models were thentrained and tested for each channel, for each algorithm and for each setof relevant hyperparameters. The performance metrics used to evaluatethe performance of a model were the Mean Absolute Error (MAE) and thecoefficient of determination R2.

The MAE is the average distance between an estimate and the targetvalue. In this case, as the estimates and targets were sets oftwo-dimensional coordinates, the Euclidean distance was the distancemetric.

The R2 is an indicator of how well future values are likely to bepredicted by the model. Values typically range from 0 to 1. A value of 1represents a model with perfect predictive power, that will yield thetarget value for any possible input value. A value of 0 represents aconstant model that always outputs the same value, regardless of theinput value. As a model can be arbitrarily bad, values can range intothe negatives.

For each color channel, the model that had achieved the highest R2 waskept as the final model. The hyperparameters used to train this modelwere saved for future use.

In one embodiment, the architecture that was settled on for thesingle-channel regressors was a combination of a Ridge Regressor and anSVM, whose outputs were averaged. Testing shown that these twoalgorithms made complimentary mistakes of the same magnitude. That is,if one overestimated the gaze position by a certain amount, the otherunderestimated the gaze position by substantially the same amount. Byaveraging their predictions, their mistakes were averaged, thus makingthe model more accurate.

For prediction fusion, i.e. the determination of the estimated gazeposition based on the respective gaze positions, all the aforementionedregression algorithms were tested in addition to linear regression.Linear regression was added as a candidate due to the very lowdimensionality of the input space for this regressor. Indeed, the inputwas comprised of the two-dimensional outputs of all three single-colorregressors, as well as the relative contribution of all three colorchannels, for a total of 9 dimensions.

Following the same approach as the single-color regressors for modelexploration and hyperparameter optimization, the linear regressionalgorithm was settled to perform color correction, as there was nosignificant gain from using a more complex regression algorithm. Thus,the method used for combination was the above-described method describedin Equation 5, where G is the final gaze estimate, We are weights, I_(c)is the illuminant value for a specific color channel, and G_(c) is thegaze estimate for a specific color channel.

G=B+Σ _(C∈[R,G,B]) W _(C) *I _(C)+Σ_(C∈[R,G,B]) W _(C) *G _(C)  (5)

The means by which the weight factors We were determined was bycomputing the relative contribution of each color channel, that is thesum of the intensity of each pixel for a given channel divided by thesum of the intensity of each pixel for each channel, as previouslydescribed.

These initial algorithms, although very quick to train, are not capableof incremental learning, which severely limits the size of the datasetthe models it is trained on, and so its ability to generalize. Testshave shown that the application required constant calibrations and theknowledge gained by calibrating with one user could not feasibly beextended to a large set of users. For these reasons, machine learningalgorithms capable of incremental learning may be preferred for a givenapplication, specifically Artificial Neural Networks, as ConvolutionalNeural Networks seemed particularly well-suited to this problem, asdescribed in details below with reference to FIGS. 15 to 20.

In one embodiment, the above-described method 10 may be embodied as acomputer program product comprising a computer readable memory storingcomputer executable instructions thereon that when executed by acomputer perform the steps of the method 10.

In one embodiment, the above-described method 10 may be embodied as asystem comprising a communication unit for at least one of receiving andtransmitting data, a memory and at least one processing unit configuredfor executing the steps of the method 10.

Referring now to FIG. 8, a system 80 for determining a gaze position ofa user in an initial image will now be described, according to oneembodiment. The system 80 is provided with an extracting unit 82, a gazeposition determining unit 84 and a gaze position estimating unit 86.

The extracting unit 82 is configured for receiving an initial image ofat least one eye of the user and extracting at least one color componentof the initial image to obtain a corresponding at least one componentimage, as detailed above. In one embodiment, the extracting unit 82 isconfigured for extracting at least two distinct color components of theinitial image to obtain at least two corresponding component images. Ina further embodiment, the extracting 82 unit is configured forextracting three distinct color components of the initial image toobtain three corresponding component images. In one embodiment, theextracting unit 82 is configured for extracting each of three RGBcomponents of the initial image, as previously described. In a furtherembodiment, the extracting unit 82 may be further configured forcropping the initial image, as described above.

The gaze position determining unit 84 is configured for receiving eachof the component images from the extracting unit 82 and determining arespective gaze position for each one of the component images, asdescribed above.

The gaze position estimating unit 86 is configured for determining anestimated gaze position in the initial image according to the respectivegaze position of each of the at least one component image and outputtingthe estimated gaze position. In the case where two or three componentimages are extracted, the gaze position estimating unit 86 is configuredfor combining each of the respective gaze positions together, forexample using weight factors, as previously detailed.

In one embodiment, the received initial image contains additionalfeatures other than the at least one eye, and the extracting unit 82 isfurther configured for identifying the at least one eye within thereceived initial image; extracting a portion of the initial imagecontaining only the at least one eye to obtain a cropped image; andextracting the at least one color component of the cropped image toobtain the corresponding at least one component image, as previouslydescribed.

In an embodiment wherein illuminant values are used, the extracting unit82 is further configured for, for each of the component images,determining an illuminant value representative of the relativecontribution of the corresponding component image to the initial image,as previously described. In this case, the gaze position estimating unit86 is further configured for combining the illuminant values with therespective gaze positions.

In an embodiment wherein head pose invariance is implemented, thereceived initial image further contains at least one facial landmark, asdetailed above. The extracting unit 82 is further configured forextracting the at least one facial landmark to obtain a corresponding atleast one landmark position. In this embodiment, the gaze positionestimating unit 86 is further configured for combining the at least onelandmark position with the respective gaze positions.

In one embodiment, each one of the units 82, 84 and 86 is provided witha respective processing unit such as a microprocessor, a respectivememory and respective communication means. In another embodiment, atleast two of the modules 82, 84 and 86 may share a same processing unit,a same memory and/or same communication means. For example, the system80 may comprise a single processing unit used by each module 82, 84 and86, a single memory and a single communication unit.

FIG. 9 is a block diagram illustrating an exemplary processing module 90for executing the steps 12 to 20 of the method 10, in accordance withsome embodiments. The processing module 90 typically includes one ormore Computer Processing Units (CPUs) and/or Graphic Processing Units(GPUs) 92 for executing modules or programs and/or instructions storedin memory 94 and thereby performing processing operations, memory 94,and one or more communication buses 96 for interconnecting thesecomponents. The communication buses 96 optionally include circuitry(sometimes called a chipset) that interconnects and controlscommunications between system components. The memory 94 includeshigh-speed random access memory, such as DRAM, SRAM, DDR RAM or otherrandom access solid state memory devices, and may include non-volatilememory, such as one or more magnetic disk storage devices, optical diskstorage devices, flash memory devices, or other non-volatile solid statestorage devices. The memory 94 optionally includes one or more storagedevices remotely located from the CPU(s) 92. The memory 94, oralternately the non-volatile memory device(s) within the memory 94,comprises a non-transitory computer readable storage medium. In someembodiments, the memory 94, or the computer readable storage medium ofthe memory 94 stores the following programs, modules, and datastructures, or a subset thereof:

An extraction module 91 for extracting at least one color component ofthe initial image to obtain a corresponding at least one componentimage;

a gaze position determining module 93 for determining the gaze positionin the component images;

a gaze position estimating module 95 for determining an estimated gazeposition in the initial image according to the respective gaze positionof each of the at least one component image;

a cropping module 97 for cropping images; and

a flattening module 99 for flattening images.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, the memory 94may store a subset of the modules and data structures identified above.Furthermore, the memory 94 may store additional modules and datastructures not described above.

Although it shows a processing module 90, FIG. 9 is intended more as afunctional description of the various features which may be present in amanagement module than a structural schematic of the embodimentsdescribed herein. In practice, and as recognized by those of ordinaryskill in the art, items shown separately could be combined and someitems could be separated.

The following description will now describe the use of deep learningalgorithms or models that may be used to improve the estimation of thegaze position in the initial image, as previously mentioned. The methodusing deep learning has similarities with the method described above;however, one notable difference is that the result of the first“primary” treatment of the distinct color component images is an“internal representation”, which is generally not the same as arespective gaze output. The internal representation has already beenmentioned above and is the output of a layer inside the neural network,to be fused with other internal representations. Normally, it has noconcrete meaning as it is not a final network output which results fromtraining and is not designed to be an estimation of any sort (it ismerely the output of that layer).

However, the method not involving neural networks that was describedabove outputs the respective gaze output in an intermediate step, andthis the respective gaze output such as the respective outputs of theRegressor R, G or B in FIG. 5, can be viewed as a specific case of the“internal representation” in which the internal representation happensto have a meaning, i.e., the respective gaze output, as it is the resultfrom training and is designed to be an intermediate estimate.

Referring now to FIG. 10, there is shown the typical structure of anartificial neuron, the fundamental unit of Artificial Neural Networks,which can be arranged in several connected layers of neurons. Theartificial neuron represents a mathematical operation applied to aweighted sum to produce an output. The artificial neuron has four maincomponents. The neuron's input is a vector IN of numbers of size N. Theneuron's weights are also a vector W_(N) of size N, multiplyingelement-wise the input vector. The neuron can have a bias term B.Finally, the neuron has an activation function f(x) which determines itsoutput, or activation a(t). The output of a neuron can thus be expressedas a(t)=ft (B+ΣIi·Wi).

FIG. 11 illustrates the structure of a fully-connected layer of neurons,which is a layer of neurons whose neurons have as an input all theoutputs of the previous layer. That is, each neuron of the layer acceptsas an input vector the entire output vector of the previous layer. Givena fully connected layer of size N and an input vector I of size M, eachneuron will have M inputs and so M weights, and so the layer has an M×Nweight matrix W and a bias vector B of size N. To simplify computations,all the neurons are made to have the same activation function. Theoutput of the layer is thus a vector given by the application of theactivation function to each element of the vector obtained by the matrixoperation I·W+B.

FIG. 12 illustrates a structure of a convolutional layer of neurons,which is a layer that takes as an input a multi-dimensional matrixinstead of a single-dimension vector. The layer is defined by itsconvolutional kernels instead of being defined by the number of neuronsit contains, as a fully-connected layer would be. These layers wereinitially designed to be used on greyscale images, but their workingprinciple can be extended to a higher dimensional input. For simplicity,we will refer to an element of the input as a pixel, but it needs onlybe an element of a matrix that may not be an image.

The workings of a convolutional layer are described here and illustratedin FIG. 12. For a given input of size H*W, a convolutional layer is saidto have H*W neurons, each associated with a pixel. The layer is alsogiven a set of M*N convolutional kernels, which are essentially a set ofweights. However, unlike the fully-connected layer in which each neuronhas its own set of weights, in a convolutional layer, all neurons sharethe same weight. Each neuron will have a receptive field on the input,of the same size as the convolutional kernels, with the neuron centeredin the receptive field. In FIG. 12 for example, we use a single 3*3kernel. The receptive fields of neurons N_(i) and N_(j) are shown.

The output of the layer is a set of feature maps, one for each kernel,of the same size as the input. Each pixel of a feature map is given bythe application of the activation function to the sum of the pixelvalues multiplied by the appropriate weight of a kernel. The result ofthis operation is the same as convolving the kernel over the input, sofiltering the input with the kernel, and applying the activationfunction to the result, hence the name “convolutional”.

FIG. 13 illustrates a structure of a convolutional stream of a neuralnetwork using fully-connected layers of neurons that can be used toimplement the method of the invention, according to one embodiment.

Primary convolutional streams are processing streams of neural networklayers that can be used to process the individual color channels of theeye images. As they are convolutional, at least one convolutional layeris included in each stream but a plurality of streams is used in oneembodiment. After a certain number of convolutional layers, a number offully-connected layers may be added downstream, although not required.In fact, it is common practice to add fully-connected layers to a set ofconvolutional layers as this tends to improve the predictive power ofthe model. For example, and without limitation, the primary stream of agiven color component image can include two or three convolutionallayers, and two or three fully-connected layers, before arriving at thefusion layer downstream, which receives the internal representation fromthe respective primary stream for this given color component image.Batch normalization method can be used on the convolutional layers,while L2 regularization and Dropout regularization method can be used onthe fully-connected layers. Other regularization methods or combinationsthereof can also be applied to these convolutional layers. It hashowever been empirically determined that the above mentioned methods arewell suited for the application. Additionally, max pooling can be usedafter each convolutional layer in order to reduce the dimensionality ofthe input to the next layer. Again, pooling is a widely used tool but isnot required. Other pooling methods may also be used, such as averagepooling. A pooling operation reduces a neighborhood of pixels to asingle value by performing some operation on the neighborhood, such asaveraging the values or taking the maximum value.

If the convolutional stream does not use fully-connected layers, theoutput of a convolutional stream is a set of feature maps, the number ofwhich corresponds to the number of kernels in the last convolutionlayer. If one or more fully-connected layers are used, the output of aconvolutional stream will be a vector containing the same number ofelements as the number of neurons in the last fully-connected layer.Additionally, if one or more fully-connected layers are used, the outputof the last convolutional layer must be flattened into a vector to beaccepted as an input by the first fully-connected layer, as previouslydescribed with reference to FIGS. 6 and 7.

FIG. 14 illustrates a structure of a fully-connected stream of a neuralnetwork that can be used to implement the method of the invention,according to another embodiment.

Primary fully-connected streams are streams of neural network layer thatcan be used to process the individual channels of the eye images. Asthey are composed exclusively of fully-connected layers, the eye imagesneed to be flattened into vector form to be accepted as inputs by thefirst fully-connected layer of the stream, as previously detailed withreference to FIGS. 6 and 7. If no fully connected-layer is used, theoutput of such a stream is the vectorized input image. Such a case maybe rare but may be useful in the case where the output of the stream isinputted into another stream for further processing. If one or morefully-connected layer is used, the output is a vector containing thesame number of elements as the number of neurons in the lastfully-connected layer.

In one embodiment, L2 regularization and Dropout regularization methodsare used on the fully-connected layers but other regularization methodsor combinations thereof can also be applied to these fully-connectedlayers.

In the case where auxiliary inputs are used, namely the illuminantvalues and the facial landmark coordinates for example, they can be feddirectly to the fusion layer, or alternatively and advantageously,auxiliary input streams of neural network can be used to apply someprocessing to the auxiliary inputs. The fusion layer will then receivethe internal representation originating from these auxiliary inputs(illuminant values and the facial landmark coordinates). Since theseinputs are of low dimensionality, being of size 3 and 16 respectively inthe previously described example, the layers used in these streams arefully-connected layers in one embodiment. If one or more fully-connectedlayers are used, the output of an auxiliary stream will be a vectorcontaining as many elements as there are neurons in the lastfully-connected layer. If no fully-connected layer is used, the outputof an auxiliary stream is its input. In one embodiment, L2regularization and Dropout regularization method or algorithm can beused on the fully-connected layers, although other methods may beconsidered. The structure of an auxiliary input stream is similar to theone of a primary fully-connected stream illustrated in FIG. 14.

As it will become more apparent below, a fusion layer is used to fusethe outputs of the upstream layers (i.e., respective internalrepresentation from the plurality of distinct primary streams andauxiliary streams) into a single vector. This is required since at leastone fully-connected layer is used to produce the output of the system,and as discussed above, a fully-connected layer accepts one and only onevector. This means that one or more fusion layers may be needed to fusethe outputs of the convolutional and auxiliary streams into a singlevector to be used as the input to the output layer.

The inputs to this layer are the outputs of at least two upstreamstreams. If no fully-connected layers are used in a convolutionalstream, the output of this stream needs to be flattened into a vectorprior to a fusion operation, as previously described.

The fusion operation itself consists in concatenating the input vectorsinto a single vector whose length is equal to the sum of the length ofall the input vectors. Fusion at this level is said to be featurefusion, as opposed to the prediction fusion used in the embodiment shownin FIG. 5. Feature fusion in a neural network can also be referred to asthe fusion of internal representations.

An internal stream of neural layers is an optional set offully-connected layers that can be used to apply further processing tothe output of a fusion layer. The input of the stream is thus the outputof a fusion layer. If one or more fully-connected layers are used, theoutput of the stream is a vector containing the same number of elementsas there are in the last fully-connected layer. If no fully-connectedlayers are used, the output of this stream is its input, so the outputof the fusion layer. The output of an internal stream can itself be usedas an input to a fusion layer. L2 regularization and Dropoutregularization method or algorithm can be used on the fully-connectedlayers, although other methods may be considered.

It should be noted that while fully-connected layers can exclusively beused in this type of stream, it is also possible to use 1D convolutionallayers instead, given the potentially relatively high dimensionality ofsome inputs. Convolutional layers however appear to be inappropriate,mostly because this type of layer is meant to exploit relationshipsbetween neighboring values, or within a neighborhood of values. Thestructure of an internal stream is similar to the one of a primaryfully-connected stream illustrated in FIG. 14.

As it will become more apparent below, in one embodiment, the output ofthe system is provided by a fully-connected layer of size one or two,depending on whether the system is to produce both X and Y gazecoordinates, or only one of these, as further described in more detailsbelow. In this embodiment, the input to this layer is either the outputof an internal stream or the output of a fusion layer.

A great many activation functions are commonly used in Artificial NeuralNetworks, and any function can be used so long as it is differentiable.Such functions include but are not limited to: the identity function,the logistic function (such as the sigmoid function), the tan h functionand the rectified linear unit (ReLU) function.

In one embodiment, the ReLU function is used for all layers except forthe output layer, which used the identity function. Such embodiment hasshown good results, but other functions may be used to yield models withdifferent performance metrics.

Referring now to FIGS. 15 to 20, a method and a system for determining agaze position of a user that rely on neural network architectures,according to some embodiments, will now be generally described in moredetails.

As it will become apparent below, in one embodiment of the method 10,the steps of determining a respective gaze position, or internalrepresentation for neural networks as it is presently the case, anddetermining an estimated gaze position are performed in combination.Indeed, the at least one component image is processed using a neuralnetwork. The neural network is implemented by one or more computers andhas one or more neural network layers. The neural network is configuredto, at run time and after the neural network has been trained, processthe at least one component image using the one or more neural networklayers to generate the estimated gaze position. Training of the neuralnetwork will be described below.

This method is implemented using the system 80 previously describedwherein the system is provided with a neural network. In thisembodiment, the neural network is configured to, at run time and afterthe neural network has been trained, process the at least one componentimage using the one or more neural network layers to generate theestimated gaze position. In one embodiment, the system 80 has at leastone primary stream forming a first portion of the neural network, eachcorresponding to a color component of the acquired images, each primarystream, each being configured to generate the respective internalrepresentation to be fused with the others, and in some cases, to bealso fused with the internal representation from auxiliary inputs suchas illuminant values and facial landmark coordinates. In other words, inthe case the three component images of an RGB image are used, the system80 has three distinct primary streams, as it will become apparent belowupon description of FIGS. 15 and 16. The system 80 also has a secondportion of the neural network, i.e., the internal stream, configured togenerate the estimated gaze position. As it should be apparent, theoutputs of the first portion of the neural network (i.e., at least oneprimary stream from the at least one color component image, and theauxiliary streams, if any) are used as the inputs of the second portionof the neural network. Various architectures for the first neuralnetworks may be used. It may comprise one or more fully-connected layersonly and/or one or more convolutional layers. If convolutional layersare used, a fully-connected layer is provided downstream the lastconvolutional layer, as detailed below. The second portion of the neuralnetwork has at least one fusion layer, each having at least onefully-connected layer. This second portion of the neural network, orinternal stream, starts from at least one of the at least one fusionlayer. The second neural network may also comprise an output layerdownstream the one or more fusion layer. The output layer may compriseone or more fully-connected layer.

Two general types of architectures will now be described with referenceto FIGS. 15 and 16, in accordance with some embodiments. Thearchitectures are only described generally since the specifics of thelayers of the neural networks fall within the domain of hyperparameteroptimization and many combinations of number of layer and layersparameters can be explored for a given architecture.

Referring now to FIG. 15, an embodiment of the system using amulti-layer perceptron will be described. This architecture containedfive fully-connected streams of neural layers, one for each input. Threeof the streams act as three distinct neural networks for the three colorchannels of the eye images, outputting a respective internalrepresentation (not a network output) at the last layer thereof. The tworemaining streams are auxiliary input streams, one for the illuminantvalues and one for the facial landmark coordinates. The outputs of thesefive streams are fused into a single vector with a fusion layer to beused as the input to an output layer. In this example, the fusion layeris comprised in the second neural network previously described.

As mentioned previously, a multi-layer perceptron is used to get anestimate of an appropriate model size, to provide a starting point to dohyperparameter optimization. In one embodiment, MLPs is chosen becausethey are much easier than ConvNets to condition properly, that is tochoose a set of hyperparameters that produce a viable model. While themodels trained under this architecture produced some viable results,MLPs are much less powerful than ConvNets on image processing problems.For this reason, ConvNets will be used in subsequent embodimentsdescribed below.

The architecture shown in FIG. 15 only contained input streams of neurallayers and a fusion layer. There was no internal stream between thefusion layer and the output layer. Additionally, the eye images usedwere of size 40×80 pixels. The same size was used in early convolutionalarchitectures, before it was increased to 80×160 pixels in an effort toimprove results.

FIG. 16 shows an embodiment of the system using convolutional neuralnetworks. Indeed, the architecture that appears to provide the bestresults uses three convolutional streams as the respective three firstneural network streams, one for each of the color channels of the eyeimages, as well as two auxiliary input streams, one for the illuminantvalues and one for the facial landmark coordinates. A single fusionlayer is used to fuse these five streams. The fusion layer is then fedinto an internal stream, and the architecture is capped by the outputlayer which produces the gaze estimate.

Some attempts to fuse the convolutional streams and the auxiliarystreams at different depths in the architecture were made, but they didnot yield better results. In these architectures, and according to oneembodiment, the convolutional streams would be fused in one fusion layerand the auxiliary streams would be fused in another. Internal streamswould then be used to process the outputs of these two fusion layers.Another fusion layer would then fuse the outputs of these internalstreams. The output of this fusion layer would be fed to a thirdinternal stream, which would finally output to the output layer.

In order to implement such architectures, a training of the neuralnetwork has to be done. The used database was composed of 2.5 millionface images, belonging to about 1500 people. The database was split intoa training set, a validation set and a test set using a 70-20-10% split.These images were obtained from volunteers tasked to look at a series ofstimuli on the screen of a mobile device of different screen sizes, beit a smartphone (such as an iPhone) or a tablet (such as an iPad). Foreach captured image, some metadata was captured which included: thedevice type, the screen size, the position of the stimulus in screencoordinates, the position of the stimulus in centimeters from thecamera, the orientation of the device (one of portrait, portrait UpsideDown, landscape Right, landscape Left), as detailed below.

In accordance with one exemplary embodiment, and without limitation,model training was performed on servers in the cloud, for instance anAmazon EC2 p3.8xlarge instance, using Keras and Tensorflow as machinelearning function libraries. Model regularization included batchnormalization on the convolutional layers, and L2 and Dropout on thefully-connected layers. The weight of the L2 regularization was 0.01 forall models. The Dropout rate was 25% for all models. These values werefound empirically and may not represent the best possible values. Thechosen architectures of the various models are given in Tables 1 to 3below. For all convolutional layers, max pooling with size 2×2 was used.To simplify hyperparameter optimization, the same architecture is usedfor all convolutional streams, and the same architecture is used forboth auxiliary streams.

Table 1 below shows the sizes of the convolutional layers. The layersizes are given in the order that they are traversed by the data, sofrom input to output. For a convolution layer, X M×N kernels means thatX number of kernels were used in this layer, with each kernel being ofsize M×N. Table 2 shows the number of layers in the auxiliary streams,and size of each layer. Table 3 shows the number of layers in theinternal stream, and size of each layer.

TABLE 1 # # Fully- Fully- Convolution Convolution connected connectedModel Layers Layer Sizes Layers Layer Sizes Portrait, 3 16 11 × 11kernels 2 200 neurons horizontal 8 5 × 5 kernels 100 neurons 4 3 × 3kernels Portrait, 3 16 11 × 11 kernels 3 200 neurons vertical 8 5 × 5kernels 100 neurons 4 3 × 3 kernels  50 neurons Portrait 3 16 11 × 11kernels 2 200 neurons Upside- 8 5 × 5 kernels 100 neurons Down, 4 3 × 3kernels horizontal Portrait 3 16 11 × 11 kernels 3 200 neurons Upside- 85 × 5 kernels 100 neurons Down, 4 3 × 3 kernels  50 neurons verticalLandscape 3 16 11 × 11kernels 2 200 neurons Right, 8 5 × 5 kernels 100neurons horizontal 4 3 × 3 kernels Landscape 3 16 11 × 11 kernels 3 200neurons Right, 8 5 × 5 kernels 100 neurons vertical 4 3 × 3 kernels  50neurons Landscape 3 16 11 × 11 kernels 2 200 neurons Left, 8 5 × 5kernels 100 neurons horizontal 4 3 × 3 kernels Landscape 3 16 11 × 11kernels 3 200 neurons Left, 8 5 × 5 kernels 100 neurons vertical 4 3 × 3kernels  50 neurons

TABLE 2 # Fully- Fully-connected Model connected Layers Layer SizesPortrait, 1 32 neurons horizontal Portrait, vertical 1 32 neuronsPortrait Upside- 1 32 neurons Down, horizontal Portrait Upside- 1 32neurons Down, vertical Landscape Right, 1 32 neurons horizontalLandscape Right, 1 32 neurons vertical Landscape Left, 1 32 neuronshorizontal Landscape Left, 1 32 neurons vertical

TABLE 3 # Fully- Fully-connected Model connected Layers Layer SizesPortrait, 3 182 neurons horizontal  91 neurons  45 neurons Portrait,vertical 2 107 neurons  53 neurons Portrait Upside- 3 182 neurons Down,horizontal  91 neurons  45 neurons Portrait Upside- 2 107 neurons Down,vertical  53 neurons Landscape Right, 3 182 neurons horizontal  91neurons  45 neurons Landscape Right, 2 107 neurons vertical  53 neuronsLandscape Left, 3 182 neurons horizontal  91 neurons  45 neuronsLandscape Left, 2 107 neurons vertical  53 neurons

In the event that the algorithms previously described does not producesufficiently accurate gaze estimates for a given application, acalibration procedure can be used during which a small dataset iscollected from the specific user to adjust the general model'spredictions.

For performing the calibration procedure, an additional set of pictureswould need to be captured. For each of these captured pictures, somestimulus would be displayed on screen, whose position (the target) wouldbe recorded and at which the user would need to look when the picture istaken. This would constitute the minimal database for the calibrationprocedure. This database could contain other metadata, such as devicetype, screen size, screen resolution and device orientation.

From there, for each captured image, the same features used by thegeneral model would be extracted from the pictures and would be fed tothe general model for processing. Here, two options are available totrain the calibration model. One option would be to capture the outputof the general model for each image. These gaze estimates wouldconstitute the inputs of the calibration model, while the true positionof the stimulus at the time of image capture would be the target. Oncetrained, such a model would be appended to the output of the generalmodel, taking it as an input and producing a new gaze coordinate. Such amodel is shown in FIG. 17.

The second option, as illustrated in FIG. 18, would be to feed thefeatures to the general model as described above, but capturing theoutput of a layer other than the output layer, so an internalrepresentation of the model, as the input to the calibration model. Thetargets for training would again be the true position of the stimulus onscreen at the time of image capture. Once trained, the calibration modelwould replace all of the layers downstream of the layer used fortraining, as illustrated.

The data collection procedure for the calibration database would involveshowing a series of stimuli to the user, while ensuring that the screenis covered entirely and evenly, as known in the art. To ensure thequality of the data, the calibration procedure should also be kept asshort as possible and should try to maximize user engagement.

Many strategies are available here. The stimuli could be made to appearat random locations throughout the screen, requiring the user to findeach stimulus before the pictures are taken. The stimuli could be madeto appear in a sequence between pairs of points on the screen, chosen atrandom, requiring the user to find the start point. The stimuli could bemade to appear in a sequence between a set of predetermined, butdisconnected pairs of points, thus making a single stimulus appear tomove along a predetermined but disconnected path. Finally, the stimulicould be made to appear in a sequence along a predetermined, continuouspath, thus creating the illusion of a single stimulus moving along saidpath. These strategies could be mixed, thus creating a calibrationprocedure during which each strategy is used for a certain amount oftime.

In one embodiment, the chosen stimulus moves along a predetermined pathwhile capturing a video of the user's face. The same effect could beachieved by capturing pictures at a certain framerate. By using thisstrategy, the user never has to find a new stimulus position after ithaving jumped, thus reducing the likelihood of bad datapoints beingcaptured while the user was looking for the stimulus. This strategy alsoallows to capture a maximum of datapoints in a set amount of time, sinceby having the stimuli “jump” from location to location, some time wouldneed to be allocated for the user to find the next stimulus to avoid theaforementioned problem. Finally, this strategy, being deterministic,allows the user to become familiar with the calibration procedure, thusincreasing the likelihood of the user following the path of the stimulusexactly.

Once the data is captured, a machine learning algorithm needs to bechosen with which the calibration models will be trained. Given therelatively low complexity of the data, these algorithms would likely bethe types of algorithms previously described, so ridge regression,decision trees, support vector machine, or even linear regression. Morecomplex algorithms like artificial neural networks could also be usedfor a specific application.

FIG. 19 illustrates a schematic of the implementation of the proposedcalibration model, according to one embodiment. The general model iscomposed of two subsystems, each of which takes in the same features andoutputs either the X or the Y gaze coordinates. These gaze positions arethen both fed to the calibration model, which is also composed of twosubsystems. Each of those subsystems takes in both gaze coordinates andoutputs either a corrected X or Y gaze coordinates.

Calibration models were then trained using support vector machines. Foreach device orientation, two calibration models were trained. Each modeltakes in the XY gaze coordinates output by the general models proper tothe appropriate device orientation, and outputs either the X or Ycorrected gaze coordinate. It would also have been possible to have asingle model outputting both gaze coordinates, but tests have shown thatthe independent determination of X or Y corrected gaze coordinateprovides better results.

Reference is now made to FIG. 20 which shows an entire system fordetermining a gaze position of a user, according to one embodiment.

For every gaze position to be estimated, the device on which the systemis installed will produce an image taken with a digital camera thatshows the face of the user, and the orientation of the device or camera,depending on the system. For example, a smartphone or tablet devicewould use the front-facing camera, and would also provide theorientation of the device, while a desktop computer would use a webcam,typically mounted on top of a screen, and would provide the orientationof the webcam.

From the initial image, five input features are extracted. Thesefeatures include the three crops of the original image that containsboth of the user eyes, or the region of the face where the eyes wouldbe. These features also include the XY coordinates of a set of faciallandmarks, and the estimated illuminant values of the initial image.

The system has four prediction streams, one for each of the fourfollowing device orientations: portrait, portrait upside-down, landscaperight and landscape left. Each of these prediction streams contains ageneral model and, if calibration has been performed for thisorientation, a calibration model. Both the general and calibrationmodels for each stream contain two subsystems. One subsystem estimatesthe horizontal gaze coordinate from the input features, while the othersubsystem estimates the vertical gaze coordinates from the samefeatures.

The predictions stream to be used is determined by the deviceorientation, which acts like a selector. The system could either haveall streams produce a gaze position estimate, with the selector beingused to select which output to use. Alternatively, the selector could beused to select which of the prediction streams should be used for agiven feature set. The latter option enables to reduce computationalcosts.

The method described herein performs particularly well for makingvarious applications involving gaze tracking for user interfaces, suchas a user interface on a smartphone, on a tablet, or on a screen of somesort. Practical application involving interactions with contentsappearing on these interfaces can be made by taking advantage of thehigh accuracy (smaller than 1 cm) that can be achieved using the presentmethod. This accuracy is notably achieved by a judicious selection ofinput images (such as a concatenation of cropped eye images with theenvironment removed). This accuracy also originates from ensuring,through the architecture as described above, that the algorithm, namelythe neural network, can adapt automatically to the illumination contextand gives a preference to the internal representation originating fromone of the color component images which gives the best results in thatillumination context. The complete separation of color component images(e.g., three color-component images of the concatenated cropped eyes)before applying a distinct neural network stream to each of them,ensures that each one is treated distinctly and can later be selectedalone for further treatment by the neural network using the mostappropriate color component image given the illumination context.

The method described herein performs particularly well when compared toother methods found in the literature, for example the study made byKrafka et al., “Eye Tracking for Everyone” from MIT, available athttp://gazecapture.csail.mit.edu/cvpr2016_gazecapture.pdf. This studyuses four inputs: each separate eye (cropped), the whole image, and abinary mask indicating face position in the image.

The present invention describes using only facial landmark coordinatesand not the whole face. In the MIT project, the first layer needsconsiderable time to be trained to identify a person's head and itsposition in the complete image. The presence in the image of theenvironment around the head is superfluous and complicates the trainingof the model. The MIT model also indicates a precision of 1.34 cm-2.12cm on mobile phones. This accuracy is not sufficient for real-lifeapplications such as the identification of keyboard elements which havea screen height or width below 1 cm. The method describes herein takesadvantage of inputs and an architecture which allow identifying thebuttons of a typical smartphone keyboard, with an accuracy in either Xor Y below 1 cm, therefore sufficient for real-life applications. Thisis at least because we have identified that using the whole image beingacquired is not useful and requires significant computational resources.In the present method, in addition to the composite image of the croppedeye images (cropped images of the eyes put together in single image)used as the input for color component images, the facial landmarkcoordinates (alone) are fed to the first layer of the network. Therequirement for computational resources is thereby reduced. Instead ofthe whole picture of the environment fed to the neural network, we usethe illuminant values as a proxy for the environmental conditions, againreducing the requirement for computational resources, both in real-timeapplication and during training. Moreover, the MIT project failed toidentify the benefit of separating RGB components of the image at theinput as described herein, which also provides technical advantages interms of accuracy when detecting edges in the eye anatomy that areuseful for gaze tracking.

The method described herein also performs particularly well whencompared to other methods found in the literature. For example, Zhang etal., available at https://arxiv.org/pdf/1504.02863.pdf, describes amethod which is only sequential, with no parallel networks. They alsoteach using only one eye, from which they lose accuracy. The method alsosolves a different problem, namely finding an eye angle, which has itsown specificities as it does not deal with head position, which needs tobe taken into account if the desired output is ax X, Y position.

The method described herein also performs particularly well whencompared to EVA Facial Mouse, a mobile application developed by Vodafonand available at http://www.fundacionvodafone.es/app/eva-facial-mouse.This application uses facial movements, not the eyes, to control themouse pointer on a device screen. This is not at all applicable to acompletely paralyzed person, who cannot move their face.

The method described herein also performs particularly well whencompared to U.S. Pat. No. 10,127,680. In this document, there is noprior training of the network. Calibration images need to be fed to thenetwork in the first place. After collecting calibration images, thenetwork is trained. Actual accuracy is expected to be very low due tovarious factors, notably the lack of training of the network. Thismethod should therefore not be expected to work in real-life conditionsas it is described therein.

The hardware necessary to perform the method includes any device capableof image acquisition, which is normally called a camera. The camera isessential as it collects the images in a proper format at a proper rateand color conditions to be fed to the analysis system. Since theanalysis system needs to be trained, an appropriate computer systemneeds to be used. This appropriate computer system is required fortraining, but may not be required for steps other than training. Actualreal-time gaze determination needs to be performed by a computer system,but the requirements for computing power can normally be met by atypical mobile device such as a smartphone or tablet of good quality.Therefore, having a computer system (not necessarily the same one as fortraining) in communication with the camera for image acquisition isessential for running the method.

Computing may be performed in various specific manners depending on thecontext. As stated above, training of the system needs a significantcomputing power to be performed, but once it is trained, the algorithmscan run on a simpler computer such as a tablet computer. However, ifcalibration needs to be done, calibration images can be advantageouslysent over a network to a remote server (or to a server in a cloudcomputing arrangement) where the calibration model can be prepared. Oncethe model is calibrated on the remote server (with a presumably moresignificant computing power than a tablet or smartphone), the calibratedmodel is sent back to the tablet or smartphone or other similar devicefor actual use of the calibrated model, locally, on the client computer.One may also contemplate performing the calibration directly on theclient computer, assuming it has enough computing power to do so andalso assuming the complete calibration model is installed thereon, inwhich case the step of sending calibrations images to a remote serverand retrieving a calibrated model can be bypassed.

The embodiments of the gaze tracking method described above can be usedfor various purposes. An example of an implementation of thegaze-tracking method described above, can involve using it in anapplication installed on an electronic device such as a smartphone,tablet and the like, for tracking the gaze of the user with respect tothe screen in order to trigger operations thereon, or collectinformation, related to what is presented on the screen.

The output of the method, i.e., X, Y coordinates with respect to areference point defined with respect from the camera, can be transformedto a screen coordinate using other inputs. For example, the relativeposition (normally fixed) between the camera and a reference point(e.g., the top left corner of the screen) should be known, as well asthe screen size and screen resolution which can be queried in the devicesettings/parameters by the mobile application installed on the device.Using these data, the X, Y output can be transformed to pixel coordinateon the screen, or any other equivalent thereof. If only an X or Y valueis needed, then this is transformed into a pixel row or column on thescreen.

Using this transformation into a screen location being looked at can beuseful to provide a way for a user to interact with the contentspresented on the screen being looked at using only eye movements. Othertypes of body movement may exist but are not required to use the methoddescribed above, as eye direction is sufficient. This is useful for auser who is paralyzed or suffers from another problem which prevents allmovements (including small facial movements) and verbal communication.Usually, a paralyzed person is able to move their eyes.

For example, on-screen elements which make up the graphical userinterface can be triggered or actuated using only the gaze, identifiedby the method as being pointing toward them. These on-screen elementscan include buttons, links, keyboard elements, and the like. Integratingthe gaze-tracking method with the larger context of electronic deviceusage can therefore ensure proper interactivity of the paralyzed personwith the screen of the electronic device, thereby using a user interfaceeffectively using their eyes only. This requires the gaze-trackingapplication to communicate the results of the tracking in terms ofscreen position to the operating system of the device or to applicationsrunning thereon, to allow real-time interactivity, as if the person wasusing a mouse pointer or tapping on a touch screen. If the method isapplied in such a context, then the use of the electronic device havinga screen becomes essential.

Other applications can also be contemplated, for example by assessingwhere on a display element of some sort the person is looking. Forexample, a camera may acquire images of a person looking at a poster orpanel and the method can be used to identify the location on the posteror panel where the person is looking. This can also apply to userinterfaces which are displayed using technologies other than a devicescreen, for example using projection or immersive environments. Themethod can therefore determine, through geometrical transformations ofthe referential (e.g., into a pixel location on the screen), that theperson is looking at displayed user-interface elements such as buttons,links, keyboard elements, and the like, on a projected image or virtualimage, and user interaction with the interface elements can then betriggered.

Although the above description relates to specific preferred embodimentsas presently contemplated by the inventors, it will be understood thatthe invention in its broad aspect includes physical and functionalequivalents of the elements described herein.

1. A method for determining a series of gaze positions of at least oneeye over time, the method comprising: capturing a video of a user's facesimultaneously with displaying a stimulus video on a screen; extractingat least one color component for each one of a plurality of imagesobtained from the video of the user's face; and based on the at leastone color component for each one of the plurality of images, determiningthe series of gaze positions of the user's face over the time of thevideo.
 2. The method of claim 1, the displaying of the stimulus videocomprises displaying stimuli at random locations throughout the screen.3. The method of claim 1, wherein the displaying of the stimulus videocomprises displaying a sequence of stimuli between a set ofpredetermined points.
 4. The method of claim 3, wherein the stimulusvideo comprises displaying the sequence of stimuli along apredetermined, continuous path.
 5. The method of claim 1, furthercomprising using the series of gaze positions over the time to train atleast one calibration model of a machine learning algorithm.
 6. Themethod of claim 5, further comprising determining another series of gazepositions based on the at least one calibration model and another videoof the user's face captured during displaying another stimulus video. 7.The method of claim 6, wherein the stimulus video comprises displaying asequence of stimuli along a predetermined, continuous path for a firsttime period and displaying another sequence of stimuli between a set ofpredetermined and disconnected pairs of points for a second time period.8. A system for determining a series of gaze positions of at least oneeye over time, the system comprising: a camera configured to capture avideo of a user's face during displaying of a stimulus video on ascreen; and a processing unit configured to: receive the video of theuser's face from the camera; extract at least one color component foreach one of a plurality of images obtained from the video of the user'sface; and based on the least one color component, determine a sequenceof gaze positions of the at least one eye.
 9. The system of claim 8,further comprising a calibration database, and the processing unit beingfurther configured to store the sequence of gaze positions andcorresponding coordinates of stimuli.
 10. A computer-implemented methodfor determining a series of gaze positions of at least one eye of auser, the method being executed by a processing unit coupled to: ascreen for displaying stimuli; a camera configured to capture a video ofa user's face during displaying of the stimuli; and the methodcomprising: determining coordinates of the stimuli to be displayed onthe screen; receiving the video of the user's face from the cameracaptured during the displaying of the stimuli; extracting at least onecolor component for each one of a plurality of images obtained from thevideo; and based on the least one color component, determining theseries of gaze positions over time of the video of the user's face. 11.The method of claim 10, wherein the method further comprises obtaining arespective internal representation based on the color components foreach one of the plurality of images.